Visual Intelligence: Toward Machine Understanding of Video Content

This paper describes progress toward developing visual intelligence algorithms (VI) that can produce humanlike text descriptions (captions) from video inputs. Video frames are assumed to be generated according to an underlying “script’ that specifies a camera model and the content and action in a scene. VI is formulated as the problem of recovering the script (or relevant portions of the script) given a sequence of video frames. Three types of scripts at different levels of abstraction are recovered: C-scripts contain object detections, poses, and descriptive information on a frame-by-frame basis; B-scripts assign persistent IDs to objects across frames and “smooth” frame-by-frame information; A-scripts provide a symbolic representation of video content using a sparse timeline in which Planning Executing Agent (PEA) graphical models (behavior snippets) are associated with agents in the scene. From the script representations, a compact text description (caption) of the action in the scene, as well as an envisionment (3D rendering) showing what the algorithm believes happened, can be generated. Scripts have been derived automatically and evaluated on a set of 240 publicly available video vignettes containing over 100,000 frames.